{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Detect Prediction Anomalies with Model Monitor\n",
    "Model Monitor captures the input, output and metadata for model predictions.  You can continuously analyze and monitor data quality.  \n",
    "\n",
    "Based on these notebooks:  \n",
    "* https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker_model_monitor/introduction/SageMaker-ModelMonitoring.ipynb\n",
    "* https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker_model_monitor/enable_model_monitor/SageMaker-Enable-Model-Monitor.ipynb\n",
    "* https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker_model_monitor/visualization/SageMaker-Model-Monitor-Visualize.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q --upgrade pip\n",
    "!pip install -q wrapt --upgrade --ignore-installed\n",
    "!pip install -q tensorflow==2.1.0\n",
    "!pip install -q transformers==2.8.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "import pandas as pd\n",
    "\n",
    "sess   = sagemaker.Session()\n",
    "bucket = sess.default_bucket()\n",
    "role = sagemaker.get_execution_role()\n",
    "region = boto3.Session().region_name\n",
    "\n",
    "sm = boto3.Session().client(service_name='sagemaker', region_name=region)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r training_job_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(training_job_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r endpoint_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(endpoint_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Configure the Data Capture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.model_monitor import DataCaptureConfig\n",
    "\n",
    "capture_s3_uri = 's3://{}/{}/model-monitor/data-capture'.format(bucket, endpoint_name)\n",
    "data_capture_config = DataCaptureConfig(enable_capture=True,\n",
    "                                        sampling_percentage=100,\n",
    "                                        destination_s3_uri=capture_s3_uri,\n",
    "                                        capture_options=[\"REQUEST\", \"RESPONSE\"],\n",
    "                                        csv_content_types=[\"text/csv\"],\n",
    "                                        json_content_types=[\"application/json\"])\n",
    "\n",
    "print(capture_s3_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Update the Endpoint to Capture Data\n",
    "This may take a few mins.  Please be patient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tensorflow.serving import Predictor\n",
    "\n",
    "predictor = Predictor(endpoint_name=endpoint_name)\n",
    "predictor.update_data_capture_config(data_capture_config=data_capture_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "\n",
    "display(HTML('<b>Review <a href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/endpoints/{}\">REST Endpoint</a></b>'.format(region, endpoint_name)))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "\n",
    "display(HTML('<b>Review <a href=\"https://s3.console.aws.amazon.com/s3/buckets/{}/{}/model-monitor/data-capture/?region={}&tab=overview\">S3 Output Data</a> Prediction Capture Data</b>'.format(bucket, endpoint_name, region)))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess.wait_for_endpoint(endpoint=endpoint_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simulate a Prediction from an Application\n",
    "With data capture enabled, we can capture the request and response payload - along additional metadata - to the S3 bucket specified above.\n",
    "\n",
    "This step invokes the endpoint with included sample data for about 2 minutes. Data is captured based on the sampling percentage specified and the capture continues until the data capture option is turned off."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RequestHandler(object):\n",
    "    import json\n",
    "    \n",
    "    def __init__(self, tokenizer, max_seq_length):\n",
    "        self.tokenizer = tokenizer\n",
    "        self.max_seq_length = max_seq_length\n",
    "\n",
    "    def __call__(self, instances):\n",
    "        transformed_instances = []\n",
    "\n",
    "        for instance in instances:\n",
    "            encode_plus_tokens = tokenizer.encode_plus(instance,\n",
    "                                                       pad_to_max_length=True,\n",
    "                                                       max_length=self.max_seq_length)\n",
    "\n",
    "            input_ids = encode_plus_tokens['input_ids']\n",
    "            input_mask = encode_plus_tokens['attention_mask']\n",
    "            segment_ids = [0] * self.max_seq_length\n",
    "\n",
    "            transformed_instance = {\"input_ids\": input_ids, \n",
    "                                    \"input_mask\": input_mask, \n",
    "                                    \"segment_ids\": segment_ids}\n",
    "\n",
    "            transformed_instances.append(transformed_instance)\n",
    "\n",
    "        transformed_data = {\"instances\": transformed_instances}\n",
    "\n",
    "        return json.dumps(transformed_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ResponseHandler(object):\n",
    "    import json\n",
    "    import tensorflow as tf\n",
    "    \n",
    "    def __init__(self, classes):\n",
    "        self.classes = classes\n",
    "    \n",
    "    def __call__(self, response, accept_header):\n",
    "        import tensorflow as tf\n",
    "\n",
    "        response_body = response.read().decode('utf-8')\n",
    "\n",
    "        response_json = json.loads(response_body)\n",
    "\n",
    "        log_probabilities = response_json[\"predictions\"]\n",
    "\n",
    "        predicted_classes = []\n",
    "\n",
    "        # Convert log_probabilities => softmax (all probabilities add up to 1) => argmax (final prediction)\n",
    "        for log_probability in log_probabilities:\n",
    "            softmax = tf.nn.softmax(log_probability)    \n",
    "            predicted_class_idx = tf.argmax(softmax, axis=-1, output_type=tf.int32)\n",
    "            predicted_class = self.classes[predicted_class_idx]\n",
    "            predicted_classes.append(predicted_class)\n",
    "\n",
    "        return predicted_classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from sagemaker.tensorflow.serving import Predictor\n",
    "from transformers import DistilBertTokenizer\n",
    "\n",
    "tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')\n",
    "\n",
    "request_handler = RequestHandler(tokenizer=tokenizer,\n",
    "                                 max_seq_length=128)\n",
    "\n",
    "response_handler = ResponseHandler(classes=[1, 2, 3, 4, 5])\n",
    "\n",
    "predictor = Predictor(endpoint_name=endpoint_name,\n",
    "                      sagemaker_session=sess,\n",
    "                      serializer=request_handler,\n",
    "                      deserializer=response_handler,\n",
    "                      content_type='application/json',\n",
    "                      model_name='saved_model',\n",
    "                      model_version=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import json\n",
    "    \n",
    "reviews = [\"This is great!\", \n",
    "           \"This is terrible.\"]\n",
    "\n",
    "predicted_classes = predictor.predict(reviews)\n",
    "\n",
    "for predicted_class, review in zip(predicted_classes, reviews):\n",
    "    print('[Predicted Star Rating: {}]'.format(predicted_class), review)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# View Captured Data\n",
    "View the captured data files in S3.  You will see files formatted by the hour per the following format:\n",
    "\n",
    "```\n",
    "s3://{destination-bucket-prefix}/{endpoint-name}/{variant-name}/yyyy/mm/dd/hh/filename.jsonl\n",
    "```\n",
    "\n",
    "_The captured data takes a few minutes to show up.  Please be patient._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(capture_s3_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_The captured data takes a few minutes to show up.  Please be patient._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "!aws s3 ls --recursive $capture_s3_uri/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_client = boto3.Session().client('s3')\n",
    "result = s3_client.list_objects(Bucket=bucket, Prefix=endpoint_name)\n",
    "capture_files = [capture_file.get(\"Key\") for capture_file in result.get('Contents')]\n",
    "\n",
    "capture_file = capture_files[0]\n",
    "print('First capture file:\\n{}'.format(capture_file))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json \n",
    "\n",
    "capture_file_contents = s3_client.get_object(Bucket=bucket, Key=capture_file).get('Body').read().decode('utf-8')\n",
    "\n",
    "json.loads(capture_file_contents)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TODO: Suggest Constraints from Baseline Dataset\n",
    "We ask SageMaker to suggest a set of baseline constraints and generate a set of summary statistics that describe these constraints.\n",
    "\n",
    "The baseline dataset must match the inference dataset schema icnluding the number of features as well as the order of features.  \n",
    "\n",
    "We will use the training dataset that we used to train the model as a suitable baseline dataset.\n",
    "\n",
    "TODO:  Header or non-header version?  With our with the label?\n",
    "\n",
    "This is the request data according to the Data Capture:\n",
    "```\n",
    "{\"captureData\":{\"endpointInput\":{\"observedContentType\":\"application/json\",\"mode\":\"INPUT\",\"data\":\"{\\\"instances\\\": [{\\\"input_ids\\\": [101, 2023, 2003, 2307, 999, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], \\\"input_mask\\\": [1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], \\\"segment_ids\\\": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}, {\\\"input_ids\\\": [101, 2023, 2003, 6659, 1012, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], \\\"input_mask\\\": [1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], \\\"segment_ids\\\": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}]}\",\"encoding\":\"JSON\"},\"endpointOutput\":{\"observedContentType\":\"application/json\",\"mode\":\"OUTPUT\",\"data\":\"{\\n    \\\"predictions\\\": [[-1.5448699, -2.04928493, -1.71272779, 0.172660828, 4.16521502], [3.47782397, -0.372900903, -1.70296526, -1.61752951, -0.441117555]\\n    ]\\n}\",\"encoding\":\"JSON\"}},\"eventMetadata\":{\"eventId\":\"e2a31d52-5685-4105-9ee1-cc28d4be7c91\",\"customAttributes\":[\"tfs-model-name=saved_model\"],\"inferenceTime\":\"2020-05-17T03:18:26Z\"},\"eventVersion\":\"0\"}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r header_train_s3_uri"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(header_train_s3_uri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "baseline_results_s3_uri = 's3://{}/{}/model-monitor/baseline-results'.format(bucket, endpoint_name)\n",
    "\n",
    "print('Baseline S3 Data: {}'.format(header_train_s3_uri))\n",
    "print('Baseline S3 Results: {}'.format(baseline_results_s3_uri))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start the Processing Job to Suggest the Constraints\n",
    "Now that we have the training data ready in S3, let's kick off a job to suggest constraints. \n",
    "\n",
    "`DefaultModelMonitor.suggest_baseline(..)` starts a SageMaker Processing Job to generate the constraints.  This processing job uses a SageMaker-provided Model Monitor Docker container with Apache Spark and the AWS Deequ open source library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sagemaker.model_monitor import DefaultModelMonitor\n",
    "from sagemaker.model_monitor.dataset_format import DatasetFormat\n",
    "from sagemaker import get_execution_role\n",
    "\n",
    "role = get_execution_role()\n",
    "\n",
    "my_default_monitor = DefaultModelMonitor(\n",
    "    role=role,\n",
    "    instance_count=1,\n",
    "    instance_type='ml.m5.xlarge',\n",
    "    volume_size_in_gb=20,\n",
    "    max_runtime_in_seconds=3600,\n",
    ")\n",
    "\n",
    "my_default_monitor.suggest_baseline(\n",
    "    baseline_dataset=header_train_s3_uri,\n",
    "    dataset_format=DatasetFormat.csv(header=True),\n",
    "    output_s3_uri=baseline_results_s3_uri,\n",
    "    wait=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# _Wait for the ^^ Processing Job ^^ to Finish_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Explore the generated constraints and statistics\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "baseline_job = my_default_monitor.latest_baselining_job\n",
    "schema_df = pd.io.json.json_normalize(baseline_job.baseline_statistics().body_dict[\"features\"])\n",
    "schema_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "constraints_df = pd.io.json.json_normalize(baseline_job.suggested_constraints().body_dict[\"features\"])\n",
    "constraints_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before proceeding to enable monitoring, you could chose to edit the constraint file as required to fine tune the constraints.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Enable Continous Monitoring\n",
    "We have collected the data above, here we proceed to analyze and monitor the data with MonitoringSchedules.\n",
    "\n",
    "## Create a schedule\n",
    "We are ready to create a model monitoring schedule for the Endpoint created earlier with the baseline resources (constraints and statistics).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sagemaker.model_monitor import CronExpressionGenerator\n",
    "from time import gmtime, strftime\n",
    "\n",
    "mon_schedule_name = 'my-monitor-schedule'\n",
    "\n",
    "report_s3_uri = 's3://{}/{}/model-monitor/reports'.format(bucket, endpoint_name)\n",
    "\n",
    "my_default_monitor.create_monitoring_schedule(\n",
    "    monitor_schedule_name=mon_schedule_name,\n",
    "    endpoint_input=predictor.endpoint,\n",
    "    output_s3_uri=report_s3_uri,\n",
    "    statistics=my_default_monitor.baseline_statistics(),\n",
    "    constraints=my_default_monitor.suggested_constraints(),\n",
    "    schedule_cron_expression=CronExpressionGenerator.hourly(),\n",
    "    enable_cloudwatch_metrics=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(report_s3_uri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "\n",
    "display(HTML('<b>Review <a href=\"https://s3.console.aws.amazon.com/s3/buckets/{}/{}/model-monitor/reports/?region={}&tab=overview\">S3 Prediction Monitoring Reports</a></b>'.format(bucket, endpoint_name, region)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Check the Schedule Status\n",
    "We created a hourly schedule above, so this will take at least one hour to appear."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "desc_schedule_result = my_default_monitor.describe_schedule()\n",
    "print('Schedule status: {}'.format(desc_schedule_result['MonitoringScheduleStatus']))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "mon_executions = my_default_monitor.list_executions()\n",
    "print(mon_executions)\n",
    "\n",
    "while len(mon_executions) == 0:\n",
    "    time.sleep(30)\n",
    "    mon_executions = my_default_monitor.list_executions()\n",
    "\n",
    "print(mon_executions)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that your monitoring schedule has been created. Please return to the Amazon SageMaker Studio to list the executions for this Schedule and observe the results going forward."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspect a specific execution (latest execution)\n",
    "In the previous cell, you picked up the latest completed or failed scheduled execution. Here are the possible terminal states and what each of them mean:\n",
    "\n",
    "* **Completed** - This means the monitoring execution completed and no issues were found in the violations report.\n",
    "* **CompletedWithViolations** - This means the execution completed, but constraint violations were detected.\n",
    "* **Failed** - The monitoring execution failed, maybe due to client error (perhaps incorrect role premissions) or infrastructure issues. Further examination of FailureReason and ExitMessage is necessary to identify what exactly happened.\n",
    "* **Stopped** - job exceeded max runtime or was manually stopped.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# _Please Wait for the ^^ Scheduled Monitor ^^ Above to Run_\n",
    "Otherwise, you will see errors below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# latest execution's index is -1, second to last is -2 and so on.\n",
    "latest_execution = mon_executions[-1] \n",
    "print(latest_execution)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "latest_job = latest_execution.describe()\n",
    "if (latest_job['ProcessingJobStatus'] != 'Completed'):\n",
    "        print(\"====STOP==== \\n No completed executions to inspect further. Please wait till an execution completes or investigate previously reported failures.\")\n",
    "        \n",
    "print(latest_job)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report_uri=latest_execution.output.destination\n",
    "print('Report Uri: {}'.format(report_uri))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List the generated reports¶\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.parse import urlparse\n",
    "s3uri = urlparse(report_uri)\n",
    "report_bucket = s3uri.netloc\n",
    "report_key = s3uri.path.lstrip('/')\n",
    "print('Report bucket: {}'.format(report_bucket))\n",
    "print('Report key: {}'.format(report_key))\n",
    "\n",
    "s3_client = boto3.Session().client('s3')\n",
    "result = s3_client.list_objects(Bucket=report_bucket, Prefix=report_key)\n",
    "report_files = [report_file.get(\"Key\") for report_file in result.get('Contents')]\n",
    "print(\"Found Report Files:\")\n",
    "print(\"\\n \".join(report_files))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Violations report\n",
    "If there are any violations compared to the baseline, they will be listed here.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "violations = my_default_monitor.latest_monitoring_constraint_violations()\n",
    "pd.set_option('display.max_colwidth', -1)\n",
    "constraints_df = pd.io.json.json_normalize(violations.body_dict[\"violations\"])\n",
    "constraints_df.head(10)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other commands\n",
    "We can also start and stop the monitoring schedules.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sm.list_monitoring_schedules()\n",
    "# sm.describe_monitoring_schedule(MonitoringScheduleName='my-monitor-schedule-2')\n",
    "# sm.delete_monitoring_schedule(MonitoringScheduleName='my-monitor-schedule-2')\n",
    "\n",
    "#my_default_monitor.stop_monitoring_schedule()\n",
    "#my_default_monitor.start_monitoring_schedule()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SageMaker Model Monitor - visualizing monitoring results\n",
    "\n",
    "The prebuilt container from SageMaker computes a variety of statistics and evaluates constraints out of the box. This notebook demonstrates how you can visualize them. You can grab the ProcessingJob arn from the executions behind a MonitoringSchedule and use this notebook to visualize the results.\n",
    "\n",
    "Let's import some python libraries that will be helpful for visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.model_monitor import MonitoringExecution\n",
    "from sagemaker.s3 import S3Downloader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Utilities for Rendering\n",
    "The functions for plotting and rendering distribution statistics or constraint violations are implemented in a utils file so let's grab that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/awslabs/amazon-sagemaker-examples/master/sagemaker_model_monitor/visualization/utils.py\n",
    "\n",
    "import utils as mu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Get Execution and Baseline details from Processing Job Arn\n",
    "\n",
    "Enter the ProcessingJob arn for an execution of a MonitoringSchedule below to get the result files associated with that execution\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "processing_job_arn = \"FILL-IN-PROCESSING-JOB-ARN\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "execution = MonitoringExecution.from_processing_arn(sagemaker_session=session.Session(), processing_job_arn=processing_job_arn)\n",
    "exec_inputs = {inp['InputName']: inp for inp in execution.describe()['ProcessingInputs']}\n",
    "exec_results = execution.output.destination"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "baseline_statistics_filepath = exec_inputs['baseline']['S3Input']['S3Uri'] if 'baseline' in exec_inputs else None\n",
    "execution_statistics_filepath = os.path.join(exec_results, 'statistics.json')\n",
    "violations_filepath = os.path.join(exec_results, 'constraint_violations.json')\n",
    "\n",
    "baseline_statistics = json.loads(S3Downloader.read_file(baseline_statistics_filepath)) if baseline_statistics_filepath is not None else None\n",
    "execution_statistics = json.loads(S3Downloader.read_file(execution_statistics_filepath))\n",
    "violations = json.loads(S3Downloader.read_file(violations_filepath))['violations']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below shows the violations and constraichecks across all features in a simple table.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mu.show_violation_df(baseline_statistics=baseline_statistics, latest_statistics=execution_statistics, violations=violations)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Distributions\n",
    "This section visualizes the distribution and renders the distribution statistics for all features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mu.show_distributions(features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = mu.get_features(execution_statistics)\n",
    "feature_baselines = mu.get_features(baseline_statistics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Execution Stats vs Baseline\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mu.show_distributions(features, feature_baselines)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
